Papers with student factorization
Structural Knowledge Distillation: Tractably Distilling Information for Structured Predictor (2021.acl-long)
Copied to clipboard
Xinyu Wang, Yong Jiang, Zhaohui Yan, Zixia Jia, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, Kewei Tu
| Challenge: | Knowledge distillation is a technique to transfer knowledge between models, typically from a large model (the teacher) to a more fine-grained one (the student). |
| Approach: | They propose a factorized form of the knowledge distillation objective for structured prediction which is tractable for many typical choices of the teacher and student models. |
| Outcome: | The proposed model is able to transfer knowledge between teacher and student models without loss of accuracy under four different scenarios. |